System and method for prompt modification based on caller hang Ups in IVRs

ABSTRACT

In a method for use in Interactive Voice Response (IVR) systems, an IVR is configured such that voice prompts provided to a caller vary systematically and caller hang-up activity is attributed to the particular version of a prompt played to the caller. Voice prompt modifications are chosen based on how long the caller willingly listens to the voice prompt. A relatively short time listening to a prompt before hang-up, indicates dissatisfaction with that prompt. The system compares caller hang-up rates for each of the phrases in a prompt and chooses the optimum solution, which is the variant of the prompt with the longest caller listening time. Optionally, the system compares hang up rates to a threshold and contingent on that comparison chooses an alternative version of the prompt.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.14/708,807, filed May 11, 2015, issued as U.S. Pat. No. 9,357,068 on May31, 2016, which is a continuation of U.S. patent application Ser. No.14/293,131, filed Jun. 2, 2014, entitled “A System and Method for PromptModification Based on Caller Hang Ups in IVRs,” issued as U.S. Pat. No.9,060,061 on Jun. 16, 2015, which is a continuation of U.S. patentapplication Ser. No. 13/911,354, filed Jun. 6, 2013, entitled “A Systemand Method for Prompt Modification Based on Caller Hang Ups in IVRs,”issued as U.S. Pat. No. 8,781,098 on Jun. 15, 2014, which is acontinuation of U.S. patent application Ser. No. 13/402,958, filed Feb.23, 2012, entitled “A System and Method for Prompt Modification Based onCaller Hang Ups in IVRs,” issued as U.S. Pat. No. 8,488,771 on Jul. 16,2013, which is a continuation of U.S. patent application Ser. No.11/732,742, filed Apr. 4, 2007, entitled “A System and Method for PromptModification Based on Caller Hang Ups in IVRs,” issued as U.S. Pat. No.8,150,020 on Apr. 3, 2012, the disclosures of which are incorporatedherein by reference.

FIELD OF THE DISCLOSURE

The invention relates generally to call centers or other call processingsystems utilizing Interactive Voice Response applications to processcalls or distribute calls among a number of service agents for handling.More particularly, the present application relates to the detecting ofuser hang ups early in the call, correlating the number of hang ups tothe number of hang ups for various versions of the same prompt or seriesof prompts, providing analysis of the hang up data such as identifyingwhere in the call users hang up, and automatically optimizingreplacement prompt selection based on hang-up data.

BACKGROUND OF THE DISCLOSURE

Interactive Voice Response (IVR) systems are computer-based telephonyapplications that answer calls from callers and, typically, attempt toautomate calls or to route calls to appropriate human agents or otherappropriate applications. Generally the system plays pre-recorded voiceprompts to which a person responds by a) pressing a number on atelephone keypad to select an option, b) speaking simple answers such as“yes”, “no”, or numbers, c) speaking keyword responses to menus, or,even d) using unconstrained natural language in an open dialog.

Automatic call distributor (ACD) systems are often the first point ofcontact when calling many larger businesses, and can be used in place ofmore expensive IVR systems. An ACD is a telephone facility that managesincoming calls and handles them based on the number called and anassociated database of handling instructions. Many companies offeringsales and service support use ACDs to validate callers, make outgoingresponses or calls, forward calls to the right party, allow callers torecord messages, gather usage statistics, balance the queues of waitingcalls to agent phone lines, and to provide other services.

IVR systems are used to create and manage services such as telephonebanking, order placement, caller identification and routing, balanceinquiry, and airline ticket booking IVR systems are generally used atthe front end of call centers to identify which service the callerwants, to retrieve numeric information such as the caller's accountnumbers, and to provide answers to simple questions such as accountbalances or pre-recorded information.

IVR systems are often criticized as being unhelpful and difficult to usedue to poor design and lack of appreciation of the callers' needs. Aproperly designed IVR system should connect callers to their desiredservice promptly and with a minimum of complexity.

When consumers access an IVR system while making a telephone call, manymay hang up the call immediately, or hang up during the first fewseconds, or a somewhat longer initial part of the call. In suchinstances a hang up could be motivated by impatience or the desire notto interact with an automated system. Often, however, a hang up iscaused by the wording of the various prompts and announcements heard.Typically, the opening verbiage in an IVR can be thought of as a seriesof announcements, messages, or phrases that come prior to the actualprompt for user input. This verbiage is often a major factor inpredicting whether a caller hangs up. When callers hang up during theinitial prompts played by the IVR, the purpose of the IVR is undermined.The process of changing various IVR system components so as to minimizethis hanging up behavior is currently completely manual.

As appreciated by those skilled in the art, tailoring prompts inapplications based on user behavior is done while the IVR is not in use.The process requires an expert such as a VI (user interface) designer orsystems analyst to examine places within prompts that exceed desiredrates of users hanging up. The manual diagnostic process of having aperson review data for initial hang-up rate as a function of promptlength is time consuming, costly, does not allow for prompt variants,and is inconvenient.

There therefore remains a need for a cost-effective technique toidentify and resolve an initial hang up as a result of voice prompt,while minimizing the above-described disadvantages.

SUMMARY OF THE INVENTION

The present invention addresses the needs described above by providing amethod and system for diagnosing and addressing caller events such ashang ups in IVR systems. One embodiment is a method for improvingperformance of an interactive voice response (IVR) system, the IVRsystem playing to callers a prompt containing a plurality of separatephrases. The method comprises the steps of measuring a caller event(e.g., hang up) rate of a test phrase in the prompt; measuring a callerevent rate of an alternative to the test phrase in the prompt; comparingthe caller event rate of the test phrase with the caller event rate ofthe alternative phrase, and replacing the test phrase contingent on thecomparison.

The steps of measuring the caller event rates may further comprise thesteps of calibrating a time of a caller event; and correlating thecaller event to a phrase that was playing at the calibrated event time.The measuring steps may further comprise normalizing the caller eventrate for a length of the phrase.

The two measuring steps may be performed simultaneously. The callerevent rates may be measured by alternatively playing the test phrase andthe alternative phrase, or by playing the phrases in random order, or byusing alternative phrases in voice prompts in a predetermined pattern.

The contingency may further include a comparison of a calculated callerevent rate to a predetermined threshold value caller event rate. Thecontingency in the method may be based on a selecting an alternativephrase with the lowest caller event rate.

The method may further comprise generating a report indicating whichphrases in a prompt have unacceptable caller event rates.

Another embodiment of the invention is a method for improving theperformance of an interactive voice response (IVR) system, the IVRsystem playing to callers a voice prompt containing a plurality ofseparate phrases. The method comprises the steps of setting a thresholdhang up rate; selecting a voice prompt containing a test phrase;measuring a hang up rate of the test phrase; comparing the hang up rateof the test phrase with the threshold hang up rate; and replacing thetest phrase contingent on the comparison.

Another embodiment of the invention is a system for improving a voiceresponse (IVR) prompt. The system comprises at least one data base ofstored voice prompts; an IVR application connected to the data base ofstored voice prompts, the IVR application configured to a play voiceprompt from the data base; a capture and tally engine configured formeasuring a number of hang ups during playing of the voice prompts bythe IVR and determining where in the call the hang up occurred; and, ananalysis engine connected to the capture and tally engine for receivingand analyzing measured hang up rates, the analysis engine furtherconnected to the IVR application for replacing the voice prompt with adifferent voice prompt from the database, based on the analysis of hangup rates.

The data base of stored voice prompts may include sequences of phrases.

The database may further comprise at least one alternative to at leastone of the phrases. The alternative may be a variant based on at leastone of content and length.

The capture and tally engine may attribute a caller hang up to a phraseof the voice prompt.

The capture and tally engine may attribute a hang up to a phrase bymeasuring how long into a voice prompt a hang up occurs.

The capture and tally engine may set a tunable parameter thatcompensates for a difference in time between a caller's actual hang uptime and the system's record of the hang up time.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a flow chart depicting a method according to one embodiment ofthe invention.

FIG. 2 is a flow chart depicting a method according to anotherembodiment of the invention.

FIG. 3 is a schematic representation of a system according to oneembodiment of the invention.

DETAILED DESCRIPTION OF THE DRAWINGS

The invention is a method and a system for modifying voice announcementsor other types of voice prompting in IVR environments such as callcenters or other communication processing applications, based on datagathered from analysis of caller hang up data. This method enablesoptimization of IVR applications so as to minimize the rate of callershanging up during the opening voice prompt.

Although the invention is described as it would apply to an IVR openingprompt, and caller hang up behavior, it could be extended to coverprompts anywhere in an IVR application and to other caller behavior,such as requests for human representatives. The disclosed techniqueswill be illustrated below in conjunction with the processing of calls inan exemplary IVR application. Those techniques, however, are not limitedto use with any particular type of environment, call center, orcommunication processing application. For example, the invention isapplicable to the processing of incoming communications, outgoingcommunications or both. The disclosed techniques can be used with ACDsystems, telemarketing systems, private-branch exchange (PBX) systems,computer-telephony integration (CTI)-based systems, as well as incombinations of these and other types of call centers. The term “callcenter” as used herein is thus intended to include any type of ACDsystem, telemarketing system or other communication system whichprocesses calls or other service requests, including voice calls, videocalls, multimedia calls, e-mail, faxes or voice messages as well asvarious combinations of these and other types of communications.

The terms “voice prompting” and “voice prompts” as used herein areintended to include any type of audibly-perceptible information,including announcements, questions, directions, statements, etc., thatmay be presented to a caller accessing an IVR application. The voiceprompt includes a series of phrases played serially to a caller. Theinitial prompt sequence refers to the first series of phrases played tothe caller. Each phrase can have alternative forms. The terms “modifiedprompt” or “modified voice prompts” refers to variants of the voiceprompt wherein a phrase or phrases differ from a base version.

FIG. 1 shows a flow diagram of a method 100 of prompt modificationaccording to one embodiment of the invention. When an IVR answers a call(step 110), the system typically begins by playing audio recordings tothe callers (step 120) consisting of greetings, announcements,instructions, menus, etc. This audio voice prompt or played announcementis composed of multiple phrases. Each phrase plays serially as in aparagraph read aloud. Although callers may experience this as if it wereone (often long) message, the initial prompt sequence is several phrasesstrung together. As an example, an application's initial prompt sequencemay be:

PI: “Acme Insurance National Claims Center.

P2: {In Spanish} For assistance in Spanish press 7.

P3: Calls are recorded and monitored.

P4: I'm your automated assistant.

P5: Please tell me the reason for your call and I'll get someone to helpyou.

The invention represents the voice prompt sequence as a series,P1-P2-P3-P4-P5, where each numbered P corresponds to a sentence orsentence fragment, also referred to as a phrase in this description.

A caller event such as a caller hang up or a request to speak to a liveperson may occur at various points in IVRs, but, typically, the largestpercentage of hang ups occurs during the initial prompt sequence, beforethe caller gives any input, be it speech or DTMF (touch tone) input. Themethod then detects (step 125) a caller event such as whether the callerdecides to exit via an early hang up. If the caller does not hang up,the application continues according to the preprogrammed IVR sequence(step 180). In the case that the caller hangs up, the method collectscaller data, including a timestamp of the hang up and stores this data(step 130) with data collected from previous caller hang ups during thisprompt.

The method proceeds to analyze the collected data (step 140), comparingthe running data from all the playings of the prompt. In (step 150) athreshold is checked to determine whether a change to a phrase iswarranted. For example, the system of the invention may be configured tochange a particular phrase if a caller event (e.g., hang up) rate forthat phrase exceeds 20%. If no change is warranted, there is no action;however, if a change is warranted then corrective action 160 is taken.For example, the best alternative phrase is chosen based on whichalternative phrase has the lowest hang up rate and the phrase is updatedwith the best alternative phrase.

FIG. 2 shows a flow diagram of a method of prompt modification 200according to a second embodiment of the invention. The system hasavailable one or more alternative phrases for the test phrase. The goalof this embodiment is to automatically evaluate each of the alternativephrases and to implement the best alternative. An advantage thisembodiment provides is the capability to play alternative phrasessequentially and in other orders, for example, the IVR answers the firstcaller and plays voice prompt alternative 1; on the second call the IVRplays voice prompt alternative 2; and so forth, playing a nextalternative voice prompt for each successive call into the IVR, up tothe number of alternatives stored; optionally the IVR may play thealternatives in any pattern. Playing the voice prompt alternatives insuch patterns, as opposed to gathering a complete data set on a firstalternative, then a second, etc., eliminates systematic errors thatmight occur from callers hearing a voice prompt played at a certain timeof day. For example, callers may be more likely to hang up when callingat lunch time than when calling at the end of the work day. Measuringthe hang up rates simultaneously eliminates such errors.

In this embodiment, when an IVR answers a call (step 210), the systemtypically begins by playing audio recordings to the callers (step 220)including greetings, announcements, instructions, menus, etc. This audiovoice prompt contains at least one test phrase. The test phrase (orphrases) is the phrase being considered for possible replacement by analternative phrase. As in the first embodiment, the voice prompt iscomposed of multiple phrases. Each phrase plays serially as in aparagraph read aloud. The method provides for discovering an optimalvoice prompt via the analysis and comparison of the hang up rates of oneor more alternative phrases to a test phrase. An alternative voiceprompt consists of multiple phrases with at least one phrase comprisinga version differing from its test counterpart.

The method continues as the first embodiment. The method (step 282)detects whether the caller decides to exit via an early hang up. If thecaller does not hang up, the application continues processing the call(step 280). In the case that the caller hangs up, the method collectscaller data and stores this data (step 230) with previous caller datacollected from hang ups during this prompt. For this embodiment data iscontinuously collected until there is a statistically significantamount.

In (step 240), the method checks whether a large enough sample has beencollected. If the sample is not large enough, the next call will beanswered using the same version of the voice prompt, or an alternativevoice prompt, depending on the pattern chosen for testing phrases (step254). When enough data has been collected, (step 250) analyzes the datafor the current test phrase. The method then implements the best testphrase 260 based on an alternative phrase having the lowest hang uprate.

When a caller hangs up during the playing of the voice prompt, thesystem collects information about the call and the phrase being playedat the time of hang-up. For example, the collected data may be atimestamp. Caller hang ups are given a timestamp in most IVRs. Thetimestamp marks the time the system records the disconnect. Becausethere is some small delay between the time a caller decides to hang upand the time the system records this as a disconnect, a calibrationcorrection, or time shift, must be applied such that a certain constantwould be subtracted from the call duration in order to map the hang upevent to the actual phrase most likely to have been playing at the timea caller hung up. The constant may, for example, compensate for systemprocessing times or may compensate for caller reaction times before hangups, or may compensate for both of these.

Over a period of time, it is possible to record and analyze thetimestamps associated with caller hang up behavior using a statisticallylarge enough sample to draw confident conclusions.

Caller hang up frequencies may correlate with prompt length and theseplots or histograms can be “normalized” into rates per unit of time. Forexample, a given prompt could be described as having a hang up rate of1% of the calls per second of play time.

This invention improves over the historical method in that it cansimultaneously monitor and trial multiple versions of a voice prompt inan IVR application. The method on initial deployment may includepredetermined variants of one of the prompts, or may include variants ofall of the prompts. In this case, prompt variants could be labeled asP1a, P1b, P1c, etc., where P1a would represent the first phrase of theprompt group and the first variant a, of that first prompt phrase. Forillustration, with the insurance application used above as an example,collected call data revealed high rates of caller hang ups at P3 and P5,so trialing alternative prompts P3a, P3b, . . . P3n and P5a, P5b . . .P5n, in various combinations and permutations may produce an effectivealternative to reduce hang ups in these phrases. The system couldinclude instructions that would cause random or systematic combinationsof the prompt variants to play and to collect the data necessary toanalyze which prompt variant produced the lowest rate of hang ups.

In one embodiment of the invention, the system may trial randomlygenerated variants of the prompts, or play each variant for a certainnumber of times. The system may generate reports of the data to aidhuman experts in analysis and selection of the prompt variant.

As noted above, the invention need not be limited to IVR prompts thatplay at the beginning of a call sequence, but could be extended toprompts that play anywhere in the IVR call flow and sequence. Also, theinvention need not be limited to caller hang up behavior, but couldinclude other behaviors to minimize or maximize. A request to speak to aperson is an example of these other behaviors.

This invention defines a way to tune the phrases in an initial voiceprompt, based upon systematic analysis of hang up patterns at thebeginning of an IVR application. The state of the art presently does notprovide a practical solution to crafting and tuning appropriate phrasesand or messages.

FIG. 3 depicts a schematic representation of a system according to oneembodiment of the invention. An exemplary system includes an IVRapplication 330 connected to at least one 310 data base as a source ofvoice prompts. The data base 310 includes stored voice prompts, eachvoice prompt comprising of a sequence of initial greeting phrases,corresponding to what a caller would hear when they are connected to anapplication. The data base 310 also includes one or more alternatives toeach voice prompt having at least one of the phrases differ from thestored voice prompts, where the alternatives can be variants based oncontent, length, etc.

IVR is also connected to an engine 350 containing logic for capturingand tallying the number of hang ups which occur during the playing ofvoice prompts, and determining where exactly in the playing of thoseannouncements each caller hangs up. That engine 350 has access to promptplaying patterns to be used in capturing data. The system would furtherhave the capability to specify a tunable parameter that defines areaction-time lag to account for the difference in time between acaller's actual hang up time and the system's record of the hang uptime.

The engine 340 is connected to the IVR, and may have the capability toprovide analysis of the hang up data, such as hang up frequency perphrase, which voice prompt results in the most hang ups in a definedperiod, etc. In addition, the engine 340 sets thresholds such asparametric values for hang up rates to aid in the determination ofwhether a phrase change is needed.

Based on periodic analysis of the hang up data, the system allowschoosing the most appropriate application messages, based on length,content, or other variants, in order to minimize the number of callerswho hang up prior to providing any input to the IVR system.

The system's output could also be modified based on other analyses ofthe responses to the greeting: for example, 3-5% of calls are normallymisdials. In this case, callers normally hang up from the systemimmediately. These quick exits could skew the results in a negative wayand thus could be treated as noise and removed from the data analysis.

The invention improves the performance of IVRs, thereby providinginformation to callers in a timely fashion by increasing the likelihoodthat callers choose to remain on the call and complete their requests.Callers will be less likely to opt out of, or hang up on, an automatedtelephone system that has been well trialed and therefore delivers welltuned voice prompts.

The described system tunes the voice prompts in the IVR based on ananalysis of when previous callers have abandoned the call. Thisinvention results in increased customer satisfaction as evidenced bydecreased hang ups and therefore decreased frustration with automatedtelephone systems. Completing the callers requests within the IVR systemsaves on the need for a human representative to handle the call.

This invention encompasses all of the needed functionality, as well asdefines viable alternatives to the initial greeting in order to reducethe number of hang-ups. This includes the ability to accurately discernwhen the caller has actually hung up, as well as capture the non-zerotime interval or length of the call associated with the hang up. Thesystem has the capability to replace existing phrases with similarphrases that differ in various aspects such as brevity, speed orintonation.

The system and method of the invention provide an accurate andtime-efficient way of enabling IVR service providers to provide higherquality service. Given the rapid increase in the use of IVR services,the method has great potential in future systems.

The method furthermore reduces the need for real-time human serviceprovider support. It also requires no special knowledge of the callersor human interface designers; for example, the data collection andanalysis are automatic. The method greatly reduces the time required toidentify and correct a voice prompt problem in an IVR service.

The foregoing Detailed Description is to be understood as being in everyrespect illustrative and exemplary, but not restrictive, and the scopeof the invention disclosed herein is not to be determined from theDetailed Description, but rather from the claims as interpretedaccording to the full breadth permitted by the patent laws. For example,while the method of the invention is described herein with respect tostandalone IVR services, the method and apparatus of the invention maybe instead embodied by call center or communications processingapplications, for example. It is further noted that the invention is notlimited to use with initial voice prompts, as described in thisspecification, but can be used with any outbound voice prompt, ACD, ortelemarketing system, or communications system existing today ordeveloped in the future. It is to be understood that the embodimentsshown and described herein are only illustrative of the principles ofthe present invention and that various modifications may be implementedby those skilled in the art without departing from the scope and spiritof the invention.

What is claimed is:
 1. A method, comprising: collecting, by a systemcomprising a processor, call data during playing to callers of a promptsequence, wherein the prompt sequence comprises a plurality of phrases,and wherein the call data comprises rates of caller events duringindividual phrases of the plurality of phrases; identifying, by thesystem, a particular one of the individual phrases based on the calldata; collecting, by the system, other call data during playing tocallers of an alternate prompt sequence comprising an alternative phrasereplacing the particular one of the individual phrases, and wherein theother call data includes a rate of caller events during the alternativephrase; comparing, by the system, a rate of caller events during theparticular one of the individual phrases with the rate of caller eventsduring the alternative phrase to obtain a comparison; and replacing, bythe system, the particular one of the individual phrases with thealternative phrase based on the comparison.
 2. The method of claim 1,wherein the playing of the prompt sequence and the playing of thealternate prompt sequence comprise a pattern of playing such that calldata collection during one of the prompt sequence and the alternateprompt sequence is not complete before initiating call data collectionduring the other of the prompt sequence and the alternate promptsequence.
 3. The method of claim 2, wherein the pattern of playingcomprises playing to alternate callers the prompt sequence and thealternate prompt sequence.
 4. The method of claim 1, further comprising:normalizing, by the system, the caller event rates for lengths of thephrases.
 5. The method of claim 1, wherein collecting call data furthercomprises collecting data comprising one of rates of caller hang ups,rates of caller requests for a human operator or both.
 6. The method ofclaim 1, wherein collecting call data further comprises collectingtimestamps of the caller events.
 7. The method of claim 6, furthercomprising: compensating, by the system, the timestamps for one of asystem processing time, a caller reaction time or both.
 8. Anon-transitory machine-readable storage medium, comprisingmachine-readable instructions that, when executed by a processor,facilitate performance of operations, comprising: collecting call dataduring playing of a prompt sequence to callers, wherein the promptsequence comprises a plurality of phrases, and wherein the call datacomprises rates of caller events during individual phrases of theplurality of phrases; identifying a particular one of the individualphrases based on the call data; collecting other call data duringplaying to callers of an alternate prompt sequence comprising analternative phrase replacing the particular one of the individualphrases, and wherein the other call data includes a rate of callerevents during the alternative phrase; comparing a rate of caller eventsduring the particular one of the individual phrases with the rate ofcaller events during the alternative phrase to obtain a comparison; andreplacing the particular one of the individual phrases with thealternative phrase based on the comparison.
 9. The non-transitorymachine-readable storage medium of claim 8, wherein the playing of theprompt sequence and the playing of the alternate prompt sequencecomprise a pattern of playing such that call data collection during oneof the prompt sequence and the alternate prompt sequence is not completebefore initiating call data collection during the other of the promptsequence the alternate prompt sequence, thereby avoiding systematicerrors due to a time of day of a call.
 10. The non-transitorymachine-readable storage medium of claim 9, wherein the pattern ofplaying comprises playing to alternate callers the prompt sequence andthe alternate prompt sequence.
 11. The non-transitory machine-readablestorage medium of claim 8, wherein the operations further comprise:normalizing the caller event rates for lengths of the phrases.
 12. Thenon-transitory machine-readable storage medium of claim 8, wherein theoperation of collecting call data further comprises collecting datacomprising one of rates of caller hang ups, rates of caller requests fora human operator or both.
 13. The non-transitory machine-readablestorage medium of claim 8, wherein the collecting of the call datafurther comprises collecting timestamps of the caller events.
 14. Thenon-transitory machine-readable storage medium of claim 13, wherein theoperations further comprise: compensating the timestamps for one of asystem processing time, a caller reaction time or both.
 15. A system,comprising: a processor; and a memory that stores executableinstructions that, when executed by the processor, facilitateperformance of operations, comprising: facilitating a collection of calldata as a prompt sequence plays to callers, wherein the prompt sequencecomprises a plurality of phrases, and wherein the call data comprisesrates of caller events during individual phrases of the prompt sequence;identifying, a particular one of the individual phrases based on thecall data; facilitating a collection of other call data during playingof an alternate prompt sequence to callers, wherein the alternate promptsequence comprises an alternate phrase replacing the particular one ofthe individual phrases, and wherein the other call data comprising arate of caller events during the alternate phrase; comparing a rate ofcaller events during the particular one of the individual phrases withthe rate of caller events during the alternate phrase to obtain acomparison; and replacing the particular one of the individual phraseswith the alternate phrase based on the comparison.
 16. The system ofclaim 15, wherein the operations further comprise playing the promptsequence and the alternate prompt sequence in a pattern of playing suchthat facilitating call data collection during one of the prompt sequenceand the alternate prompt sequence is not complete before facilitatingcall data collection during the other of the prompt sequence and thealternate prompt sequence.
 17. The system of claim 16, wherein thepattern of playing comprises playing to alternate callers the promptsequence and the alternate prompt sequence.
 18. The system of claim 15,wherein the operations further comprise: normalizing the caller eventrates for lengths of the phrases.
 19. The system of claim 15, whereinthe collection data comprises one of rates of caller hang ups, rates ofcaller requests for a human operator or both.
 20. The system of claim15, wherein the operations further comprise collecting timestamps of thecaller events.